ETSITS 186 008-3 V1.1.1 



(2007-10) 



Technical Specification 



Telecommunications and Internet converged Services and 

Protocols for Advanced Networking (TISPAN); 

IMS/NGN Performance Benchmark 

Part 3: Traffic Sets and Traffic Profiles 




ETSI TS 186 008-3 V1.1.1 (2007-10) 



Reference 



DTS/TISPAN-06024-3-NGN 
Keywords 



IMS, performance, service 



ETSI 

650 Route des Lucioles 
F-06921 Sophia Antipolis Cedex - FRANCE 

Tel. : +33 4 92 94 42 00 Fax: +33 4 93 65 47 1 6 

Siret N ° 348 623 562 0001 7 - NAF 742 C 
Association a but non lucratif enregistree a la 
Sous-Prefecture de Grasse (06) N° 7803/88 



Important notice 



Individual copies of the present document can be downloaded from: 
http://www.etsi.orq 

The present document may be made available in more than one electronic version or in print. In any case of existing or 

perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF). 

In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive 

within ETSI Secretariat. 

Users of the present document should be aware that the document may be subject to revision or change of status. 

Information on the current status of this and other ETSI documents is available at 

http://portal.etsi.orq/tb/status/status.asp 

If you find errors in the present document, please send your comment to one of the following services: 

http://portal.etsi.orq/chaircor/ETSI support.asp 

Copyright Notification 

No part may be reproduced except as authorized by written permission. 
The copyright and the foregoing restriction extend to reproduction in all media. 

© European Telecommunications Standards Institute 2007. 
All rights reserved. 

DECT'^", PLUGTESTS™ and UMTS™ are Trade IVlarks of ETSI registered for the benefit of its IVIembers. 
TIPHON^" and the TIPHON logo are Trade Marks currently being registered by ETSI for the benefit of its Members. 
2QppTM |g g jracle Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. 



ETSI 



ETSI TS 186 008-3 V1.1.1 (2007-10) 



Contents 



Intellectual Property Rights 4 

Foreword 4 

Introduction 4 

1 Scope 5 

2 References 5 

2.1 Normative references 6 

3 Definitions and abbreviations 6 

3.1 Definitions 6 

3.2 Abbreviations 8 

4 Benchmark tests 9 

4.1 Benchmark test goals 9 

4.2 Initial benchmark traffic set and traffic-time profile 9 

4.3 Initial benchmark test implementation 15 

4.3.1 SUT Configuration 15 

4.3.2 Preamble 15 

4.3.3 Test Execution 15 

4.3.4 Graphs 16 

Annex A (informative) : Bibliography 17 

History 18 



£75/ 



ETSI TS 186 008-3 V1.1.1 (2007-10) 



Intellectual Property Rights 



IPRs essential or potentially essential to the present document may have been declared to ETSI. The information 
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found 
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in 
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web 
server ( http://webapp.etsi.org/IPR/home.asp ). 

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee 
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web 
server) which are, or may be, or may become, essential to the present document. 



Foreword 

This Technical Specification (TS) has been produced by ETSI Technical Committee Telecommunications and Internet 
converged Services and Protocols for Advanced Networking (TISPAN). 

The present document is part 3 of a multi-part deliverable covering the IMS/NGN Performance Benchmark, as 
identified below: 

Parti: "Core Concepts"; 

Part 2: "Subsystem Configurations and Benchmarks". 

Part 3: "Traffic Sets and Traffic Profiles". 



Introduction 



TS 186 008-1 [1] provides a general introduction to the environment in which the benchmark exists. 

TS 186 008-2 [2] documents the subsystem configurations, use-cases and design objectives corresponding to them. 

The present document defines an initial benchmark tests through definitions of traffics sets and traffic profiles. 
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Scope 



The present document is for an initial, revision 1 .0, release of an IMS/TISPAN NGN performance benchmark. The 
metrics measured and reported are for performance of this subsystem under a communications application load. 

The benchmark is defined for the IMS network as a whole, as well as for several subsystems of an IMS network. The 
benchmark is designed so that nodes composing a subsystem can also be benchmarked alone. 

This multi-part deliverable consists of three parts. TS 186 008-1 [1] contains overall benchmark descriptions, 
architectures, processes, and information models that are common to all specific benchmarking scenarios. Part 1 of this 
document has no attachments. TS 186 008-2 [2] contains the IMS and ETSI TISPAN SUT subsystem configurations, 
the specific benchmarking use-cases and scenarios, along with scenario specific metrics and design objectives. It also 
defines the SUT configuration parameters. The present document defines an initial benchmark test through the 
specification of a traffic set, traffic-time profile, and benchmark test procedure. 

The initial benchmark test data defined in the present document include: 

• Traffic set. 

• Traffic-time profile. 

• Benchmark test procedure. 

It is expected that the present document will be updated as service providers refine their use of the benchmark and their 
deployment of IMS and ETSI TISPAN networks. 



References 



References are either specific (identified by date of publication and/or edition number or version number) or 
non-specific. 

• For a specific reference, subsequent revisions do not apply. 

• Non-specific reference may be made only to a complete document or a part thereof and only in the following 

cases: 

if it is accepted that it will be possible to use all future changes of the referenced document for the 
purposes of the referring document; 

for informative references. 

Referenced documents which are not found to be publicly available in the expected location might be found at 
http://docbox.etsi.org/Reference . 

For online referenced documents, information sufficient to identify and locate the source shall be provided. Preferably, 
the primary source of the referenced document should be cited, in order to ensure traceability. Furthermore, the 
reference should, as far as possible, remain valid for the expected Ufe of the document. The reference shall include the 
method of access to the referenced document and the full network address, with the same punctuation and use of upper 
case and lower case letters. 

NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee 
their long term validity. 
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2.1 Normative references 



The following referenced documents are indispensable for the application of the present document. For dated 
references, only the edition cited applies. For non-specific references, the latest edition of the referenced document 
(including any amendments) applies. 

[1] ETSI TS 186 008-1: "Telecommunications and Internet converged Services and Protocols for 

Advanced Networking (TISPAN); IMS/NGN Performance Benchmark Part 1: Core Concepts". 

[2] ETSI TS 186 008-2: "Telecommunications and Internet converged Services and Protocols for 

Advanced Networking (TISPAN); IMS/NGN Performance Benchmark Part 2: Subsystem 
Configurations and Benchmarks". 



Definitions and abbreviations 



3.1 Definitions 

For the purposes of the present document, the following terms and definitions apply: 

arrival distribution: function describing the probability distribution of the interarrival time of events (e.g. messages) 

Attempts Per Second (APS): rate at which scenarios are executed. Depending on the scenario, the actual 
implementation might be RAPS, Registration Attempts Per Second, or CAPS, Call Attempts Per Second 

background load: workload applied to an SUT during a benchmark test, for the purpose of consuming SUT resources 
during a benchmark test and changing the traffic intensity at which the capacity of the SUT is reached 

benchmark log: data file containing measurements of SUT performance collected during the execution of a test 
procedure 

benchmark report: documented generated at the conclusion of a test procedure containing the metrics measured 
during the execution of the test and/or computed from the data collected in the benchmark log 

benchmark test: aprocedure by which a test system interacts with a System Under Test to measure its behavior and 
produce a benchmark report 

configuration: specification of a subset of IMS architectural elements and metrics for which collection of benchmark 
tests can be defined 

design objective: probabilistic model of delay and failure requirements for an SUT, associated with a use-case. It is 
specified by threshold values and probabilities for delay and scenario failure 

Design Objective Capacity (DOC): largest load an SUT can sustain while not exceeding design objectives defined for 
a use-case 

DOC overload: condition, or part of a load profile, in which the system load exceeds the DOC 

DOC underload: condition, or part of a load profile, in which the system load does not exceed the DOC 

duration distribution: function (e.g. Poisson) describing the probability distribution of the duration of an event 
(e.g. a call) 

inadequately handled scenario attempt: scenario attempt which either fails or which exceeds the threshold values 
defined for the use case of which the scenario attempt is an instantiation 

maximum capacity: smallest scenario arrival rate at which the successful scenario rate cannot be increased 

metric: performance measurement of an SUT reported in a benchmark report 

parameter: attribute of an SUT, test system, system load, or traffic set whose value is set externally and prior to a 
benchmark test, and whose value affects the behavior of the benchmark test 
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percent registered subscribers: parameter specifying the percent of subscribers with records in the HSS/UPSF that are 
active during the benchmark test 

percent roaming subscribers: parameter specifying the percent of active subscribers who are roaming 

preamble: phase at the beginning of a test procedure during which initialization of the test system and System Under 
Test is performed 

protocol diagram: diagram depicting a collection of architectural elements as vertical lines, and protocol interactions 
between the elements as directed lines between architectural elements, where the vertical order in which the directed 
lines appear indicate time sequence 

recovery capacity: when a traffic-time profile describes a scenario arrival rate starting at a value greater than maximum 
capacity and monotonically decreasing, the maximum value at which design objectives for the use-case in effect are no 
longer exceeded 

SAPS increase amount: increment by which the average SAPS changes between steps of a profile. Equivalent to 
system load increase amount 

scenario: specific path through a use-case, including the sequence of messages exchanged by all agents, and (when 
meaningful) a scenario duration distribution (e.g. the duration of a call) 

EXAMPLE: An example of a scenario is "simple call - succeeded". 

scenario attempt: event of a scenario being initiated by the test system and handled by the SUT 

scenario attempts per second: average number of scenarios that are instantiated by the test system per second 

scenario arrival distribution: probability distribution that governs the arrival times of scenarios during a test phase 

scenario duration distribution: probability distribution that governs the duration of an individual scenario 

scenario percent of system load: relative frequency of an individual scenario within a system load 

simultaneous scenarios: number of scenarios that the test system may allow a single UE to perform simultaneously 

step number: for a profile consisting of a sequence of steps, the number of steps 

step time: length of time, in a profile consisting of a sequence of steps, at which the average scenario arrival rate 
remains at the same value 

step transient time: parameter representing the time interval, measured from the beginning of a step, for which counts 
of scenario attempts and inadequately handled scenario attempts are not kept 

stir time: parameter representing a period of time in the preamble of a benchmark test in which a system load is run in 
order to allow initial transient conditions attenuate to an insignificant level 

subscriber base: information elements that describe simulated users 

system load: stream of protocol interactions presented to the SUT by the test system 

system load increase amount: increment by which the average SAPS changes between steps of a profile. Equivalent to 
SAPS increase amount 

System Under Test (SUT): collection of hardware and software whose performance is measured by the benchmark test 

test parameters: parameters whose values determine the behavior of a benchmark test 

test procedure: specification of the steps to be performed by a benchmark test 

test scenario: specific path through a use-case, whose implementation by a test system creates a system load 

test system: collection of hardware and software which presents a system load to a System Under Test and collects data 
on the System Under Test"s performance, from which metrics can be computed 

total provisioned subscribers: number of simulated subscribers with records in the HSS/UPSF 
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traffic-time profile: evolution of the average scenario arrival rate over time. It is specified by a scenario arrival 
distribution and a function of average scenario arrival rate as a function of time 

traffic set: mixture of scenarios whose proportional contributions to traffic are fixed 

use-case: specification of a type of interaction between a test system and a system under test, corresponding to a mode 
of end-user behavior 

NOTE: A use-case is a collection of scenarios. 

EXAMPLE: An example of a use-case is "simple call", which may contain scenarios "simple call - succeeded", 
"simple call - no answer", and others. 

user behavioral model: model that defines the number and rate at which an individual user of an IMS system makes 
scenario attempts 



3.2 



Abbreviations 



For the purposes of the present document, the following abbreviations apply: 

%IHS Percent Inadequately Handled Scenarios 

3GPP Third Generation Partnership Project 

3GPP2 Third Generation Partnership Project-2 

APS Attempts Per Second 

AS Application Server 

ATCA Advanced Telecom Computing Architecture 

BGCF Border Gateway Control Function 

CDMA Code Division Multiple Access 

CSCF Call Session Control Function 

DO Design Objective 

DOC Design Objective Capacity 

GSM Global System for Mobile communication 

HSS Home Subscriber Server 

IBCF Interconnection Border Control Function 

I-BGF Interconnect-Border Gateway Function 

I-CSCF Interrogating-CSCF 

IHSA Inadequately Handled Scenario Attempt 

IMS IP Multimedia Subsystem 

ISDN Integrated Services Digital Network 

IWF: Inter- Working Function 

JAIN Java API for Integrated Networks 

LA N Local Area Network 

MEM MEMory usage 

MRFC Media Resource Function Controller 

MRFP Media Resource Function Processor 

NGN Next Generation Networks 

OMG Object Management Group 

PCI Peripheral Component Interconnect 

P-CSCF Proxy-CSCF 

PSTF Public Switched Telecommunications Network 

QoS Quality of Service 

SBC Session Border Control 

S-CSCF Serving CSCF 

SGF Serving Gateway Function 

SIMS SIMultaneous Sessions (per user) 

SLF Subscriber Location Function 

SUT System Under Test 

TEM Telecommunications Equipment Manufacturer 

TISPAN Telecommunications and Internet convergence Services for Advanced Networking 

T-MGF Trunk Media Gateway Function 

TRT Total Round-trip Time 

UE User Equipment 
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SIPP 
UPSF 



Simple Internet Protocol Plus 
User Profile Server Function 



4 Benchmark tests 

TS 186 008-1 [1] and TS 186 008-2 [2] have defined the framework for defining and execudng an IMS/NGN 
performance benchmark. The present document specifies a benchmark test, which may be implemented and performed 
as-is, or which may serve as an example for future benchmark tests developed by a service provider or SUT 
implementor. As experience with deployed IMS/NGN systems and with this benchmark is developed, additional 
benchmark tests will be added to future releases of this multi-part deliverable. 



4.1 Benchmark test goals 



A benchmark test may be used either for comparison (e.g. comparing the performance of two products), or for 
prediction (e.g. the configuration specified for a benchmark test is similar enough to a service provider"s requirements 
that the result of the test can be used as an estimate of the performance of their deployed system). 

In order to accomplish these goals, a suite of benchmark tests must cover a range of scenarios that are representative of 
the real world. Given the early nature of the IMS/NGN deployments, the data required to construct these scenarios is 
still in the process of being collected. For that reason, the benchmark test described here is more useful for prediction 
than for comparison. 

It is expected that the benchmark test specified in this part of the standard will have to be supplemented by additional 
benchmark tests that follow the framework of the standard, but which are parameterized differently. 

Additionally, while the primary metric collected by this benchmark test is the Design Objective Capacity (DOC) of the 
SUT, as experience is gained both with IMS/NGN deployments and with this benchmark, additional metrics will be 
defined. 

4.2 Initial benchmark traffic set and traffic-time profile 

As described in detail in TS 186 008-1 [1], a benchmark test measures the behavior of an SUT for a specified traffic set 
and traffic-time profile. A traffic set is composed of a mixture of test scenarios, whose relative frequency of occurrence 
is specified by traffic set parameters; the traffic-time profile is a specification of how the average arrival rate of test 
scenarios evolves over the execution of the benchmark test. 

Tables 1 and 2 represent an educated assessment on the part of the release 1 benchmark authors of a generic IMS/NGN 
traffic set and profile covering the three major use-cases defined in TS 186 008-2 [2], clause 3. 

In the execution of the test, test scenario types are randomly selected according to the frequencies listed in the table. 
Thereafter, users are randomly selected from the database of subscribed users, and the information associated with them 
is used to fill the fields of the messages generated by the test system. 

In future releases of this multi-part deliverable, a user behavioral model may be specified, and the procedure for 
selecting a user and a test scenario type may be changed. This topic is discussed in TS 186 008-1 [1]. 
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Table 1 : Initial benchmark traffic set 
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Use Case 
Section 


Test Scenario 


Scenario ID 


Type 


Scenario 

%of 
System 

Load 


Scenario 

Arrival 

Distribution 


Scenario 

Duration 

Distribution 

(calls), 
message size 

(text 
messaging) 


TS 186 008-2 [2], 
clause 5.1: 
Registration/ 
De-registration 
Use-Case 


TS 186 008-2 [2], clause 5.1.2.1: Scenario 1 - 
Successful Initial Registration without 
Synchronization 


PX_S1_1 


float 


1 % 


Poisson, 
mean 

selected by 
traffic-time 
profile 




TS 186 008-2 [2], clause 5.1.2.2: Scenario 2 - 
Successful Initial Registration with 
Synchronization 


PX_S1_2 


float 


0% 






TS 186 008-2 [2], clause 5.1.2.3: Scenario 3 - 
Re-Registration - User Currently Registered 


PX_S1_3 


float 


1 % 


Poisson, 
mean 

selected by 
traffic-time 
profile 




TS 186 008-2 [2], clause 5.1.2.4: Scenario 4 - 
Re-Subscription - User Currently Registered 


PX_S1_4 


float 


0% 






TS 186 008-2 [2], clause 5.1 .2.5: Scenario 5 - 
Re-Registration - User Roaming 


PX_S1_5 


float 


0% 






TS 186 008-2 [2], clause 5.1 .2.6: Scenario 6 - 
UE Initiated De-Registration 


PX_S1_6 


float 


1 % 






TS 186 008-2 [2], clause 5.1 .2.7: Scenario 7 - 
Network Initiated De-Registration 


PX_S1_7 


float 


0% 






TS 186 008-2 [2], clause 5.1.2.8: Scenario 8 - 
Network initiated de-registration upon roaming or 
expiration 


PX_S1_8 


float 


0% 






TS 186 008-2 [2], clause 5.1.2.9: Scenario 9 - 
Network initiated re-authentication 


PX_S1_9 


float 


0% 







£75/ 



11 



ETSI TS 186 008-3 VI. 1.1 (2007-10) 



Use Case 
Section 


Test Scenario 


Scenario ID 


Type 


Scenario 

%of 
System 

Load 


Scenario 

Arrival 

Distribution 


Scenario 

Duration 

Distribution 

(calls), 
message size 

(text 
messaging) 


TS 186 008-2 [2], 
clause 5.2: 
Session Set-Up/ 
Tear-Down 
Use-Case 
PX_ 


TS 186 008-2 [2], clause 5.2.2.1 : Scenario 1 - 
Successful Call - Resource reservation on both 
sides 


PX_S2_1 


float 


12% 


Poisson, 
mean 

selected by 
traffic-time 
profile 


Exponential, 
mean 120 s 


TS 186 008-2 [2], clause 5.2.2.2: Scenario 2 - 
Successful Call - No resource reservation on 
originating side 


PX_S2_2 


float 


12% 


Poisson, 
mean 

selected by 
traffic-time 
profile 


Exponential, 
mean 120 s 


TS 186 008-2 [2], clause 5.2.2.3: Scenario 3 - 
Successful Call - No resource reservation on the 
terminating side 


PX_S2_3 


float 


12% 


Poisson, 
mean 

selected by 
traffic-time 
profile 


Exponential, 
mean 120 s 


TS 186 008-2 [2], clause 5.2.2.4: Scenario 4 - 
Successful Call - No resource reservation on 
either side 


PX_S2_4 


float 


12% 


Poisson, 
mean 

selected by 
traffic-time 
profile 


Exponential, 
mean 120 s 


TS 186 008-2 [2], clause 5.2.2.5: Scenario 5 - 
Successful Call - Resource reservation on 
originating side and non-IMS terminating side 


PX_S2_5 


float 


0% 






TS 186 008-2 [2], clause 5.2.2.6: Scenario 6 - 
Successful Call - No resource reservation on 
originating side and non-IMS terminating side 


PX_S2_6 


float 


0% 






TS 186 008-2 [2], clause 5.2.2.7: Scenario 7 - 
Successful Call - Originating side non-IMS and 
resource reservation on terminating side 


PX_S2_7 


float 


0% 






TS 186 008-2 [2], clause 5.2.2.8: Scenario 8 - 
Successful Call - Originating side non-IMS and 
no resource reservation on terminating side 


PX_S2_8 


float 


0% 
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Test Scenario 



TS 186 008-2 [2], clause 5.2.2.9: Scenario 9 - 
Abandoned Call - Resource reservation on both 
sides 



TS 186 008-2 [2], clause 5.2.2.10: Scenario 10 
Abandoned Call - No resource reservation on 
originating side 



TS 1 86 008-2 [2], clause 5.2.2.1 1 : Scenario 1 1 
Abandoned Call - No resource reservation on 
terminating side 



TS 186 008-2 [2], clause 5.2.2.12: Scenario 12 
Abandoned Call - No resource reservation on 
either side 



TS 186 008-2 [2], clause 5.2.2.13: Scenario 13 
Abandoned Call - Resource reservation on 
originating side and non-IMS terminating side 



TS 186 008-2 [2], clause 5.2.2.14: Scenario 14 
Abandoned Call - No resource reservation on 
originating side and non-IMS terminating side 



TS 186 008-2 [2], clause 5.2.2.15: Scenario 15 - 
Abandoned Call - Originating side non-IMS and 
resource reservation on terminating side 



TS 186 008-2 [2], clause 5.2.2.16: Scenario 16 - 
Abandoned Call - Originating side non-IMS and 
no resource reservation on terminating side 



Scenario ID 



PX S2 9 



PX S2 10 



PX S2 11 



PX S2 12 



PX S2 13 



PX S2 14 



PX S2 15 



PX S2 16 



Type 



float 



float 



float 



float 



float 



float 



float 



float 



Scenario 

%of 
System 

Load 



3% 



3% 



3% 



3% 



0% 



0% 



0% 



0% 



Scenario 

Arrival 

Distribution 



Poisson, 
mean 

selected by 
traffic-time 
profile 



Poisson, 
mean 

selected by 
traffic-time 
profile 



Poisson, 
mean 

selected by 
traffic-time 
profile 



Poisson, 
mean 

selected by 
traffic-time 
profile 



Scenario 

Duration 

Distribution 

(calls), 
message size 

(text 
messaging) 



Exponential, 
mean 15 s 



Exponential, 
mean 15 s 



Exponential, 
mean 15 s 



Exponential, 
mean 15 s 
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TS 186 008-2 [2], clause 5.2.2.19: Scenario 19 
Rejected Call - No resource reservation on 
terminating side 



Test Scenario 



TS 186 008-2 [2], clause 5.2.2.17: Scenario 17 - 
Rejected Call - Resource reservation on both 
sides 



TS 186 008-2 [2], clause 5.2.2.18: Scenario 18 
Rejected Call - No resource reservation on 
originating side 



TS 186 008-2 [2], clause 5.2.2.20: Scenario 20 
Rejected Call - No resource reservation on 
either side 



TS 186 008-2 [2], clause 5.2.2.21 : Scenario 21 
Rejected Call - Resource reservation on 
originating side and non-IMS terminating user 



TS 186 008-2 [2], clause 5.2.2.22: Scenario 22 
Rejected Call - No resource reservation on 
originating side and non-IMS terminating side 



TS 186 008-2 [2], clause 5.2.2.23: Scenario 23 - 
Rejected Call - Originating side non-IMS and 
resource reservation on terminating side 



TS 186 008-2 [2], clause 5.2.2.24: Scenario 24 - 
Rejected Call - Originating side non-IMS and no 
resource reservation on terminating side 



TS 186 008-2 [2], clause 5.2.2.25: Scenario 25 
Failed Call 



Scenario ID 



PX S2 17 



PX S2 18 



PX S2 19 



PX S2 20 



PX S2 21 



PX S2 22 



PX S2 23 



PX S2 24 



PX S2 25 



Type 



float 



float 



float 



float 



float 



float 



float 



float 



float 



Scenario 

%of 
System 

Load 



3% 



3% 



3% 



3% 



0% 



0% 



0% 



0% 



1 % 



Scenario 

Arrival 

Distribution 



Poisson, 
mean 

selected by 
traffic-time 
profile 



Scenario 

Duration 

Distribution 

(calls), 
message size 

(text 
messaging) 
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Use Case 


Test Scenario 


Scenario ID 


Type 


Scenario 


Scenario 


Scenario 


Section 








%of 

System 

Load 


Arrival 
Distribution 


Duration 
Distribution 

(calls), 
message size 

(text 
messaging) 


TS 186 008-2 [2], 


TS 186 008-2 [2], clause 5.3.2.1 : Scenario 1 - 


PX S3 1 


float 


19% 


Poisson, 


message size 


clause 5.3: 


Successful Message Exchange 








mean 


uniform 


Page-mode 










selected by 


[0 to 140] 


Messaging 










traffic-time 


characters 


Use-Case 










profile 




TS 186 008-2 [2], clause 5.3.2.2: Scenario 2 - 


PX S3 2 


float 


5% 


Poisson, 


message size 




Unsuccessful Message Exchange - Called User 








mean 


uniform 




Not Found 








selected by 

traffic-time 

profile 


[0 to 140] 
characters 
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Table 2: Initial benchmark traffic-time profile 



Traffic-time profile Parameter 


Traffic-time profile 
Value 


Notes 


PX SimultaneousScenarios (SIMS) 


2 


Maximum per UE 


PX TotalProvisionedSubscribers 


100,000 Subs 


Data in Part 2 


PX_PercentRegisteredSubscribers 


40% 


At test start. Tlie percent of registered subscribers will 
fluctuate during the test 


PX PercentRoamingSubscribers 


None 


No roaming in Release 1 


PX StepNumber 


3 steps 


DOC underload, DOC, and DOC overload 


PX StepTransientlime 


1 20 seconds 


Maximum 


PX StepTime 


30 minutes 


Minimum 


PX BackgroundLoad 


None 




PXSApSlncreaseAmount 


lOSApS 


Maximum 

Report three results, step before, DOC and step after 


PX SystemLoad 


DOC 


Reported result in Scenario Attempts Per Second 


PX_IHS % InAdequately Handle 
Scenario Attempts Maximum (IHS) 


0,1 % 


Average over a test step 



4.3 Initial benchmark test implementation 

The specification does not dictate the specific implementation of a test scanario. The test scenarios are defined in 
TS 186 008-2 [2] as protocol diagrams. These scenarios are implemented by either a commercial test system provider or 
as part of a benchmark test run. Example implementations include using the ETSI TTCN3 notation, using an XML 
notation (e.g. based on the open source SIPP), or specifically coding the test in a general programming language. For 
comparison (and ultimately certification) purposes, a specification of the test system used to implement the traffic-time 
profile, and documentation of the test scenario implementation in the test system with sufficient detail to be 
independently replicated, shall be included as part of the report. 

4.3.1 SUT Configuration 

The initial benchmark supports a Session Control Subsystem SUT configuration as defined in TS 186 008-2 [2] clause 
4. Release 1 does not specify reliability or availability requirements. The availability architecture and design objective 
target of the SUT should be described in the test report. 

4.3.2 Preamble 

The preamble defines the steps necessary to configure the SUT for a benchmark run. The following steps must be 
completed before the initial benchmark test run. 

1) The SUT must be started from a cold boot. 

2) "Total Provisioned Subscribers" must be provisioned in the database. 

3) The initial "Percent Registered Subscribers" must be setup. 

4.3.3 Test Execution 

The initial benchmark test must contain StepNumber stair steps in the profile. 

The test execution is valid if the profile has steps in the DOC underload range and at least one step in the DOC overload 
range. 
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4.3.4 Graphs 

The following graphs shall be plotted in the benchmark report: 

• Scenario success rate: 

X-Axis: time (s). 

Y-Axis 1 : Scenario Attempts Per Second for the traffic set. 

Y-Axis 2: Percentage of Inadequately Handled Scenario Attempts. 

• Scenario average transaction response time {for each identified scenario in the traffic set}: 

X-Axis: time (s). 

Y-Axis 1 : Scenario Attempts Per Second for the individual scenario. 

Y-Axis 2: For each TRTdesign objective in the identified scenario, SUM of the TRT for a second 
divided by the SApS for the second. 

• Scenario Retransmissions {for each identified scenario in the traffic set that has retransmissions}: 

X-Axis: time (s). 

Y-Axis 1 : Scenario Attempts Per Second. 

Y-Axis 2: Number of retransmissions in a second for that scenario. 

• CPU {on each of SUT node}: 

X-Axis: time (s). 

Y-Axis 1 : Scenario Attempts Per Second. 

Y-Axis 2: CPU. 

• MEM { on each of SUT node } : 

X-Axis: time (s). 

Y-Axis 1 : Scenario Attempts Per Second. 

Y-Axis 2: MEM. 
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